Pinterest的技术架构

Pinterest的技术架构

本文来自QCon
Scaling
Pinterest
Marty Weiner
Cloud Ninja
Yash Nelapati
Ascii Artist
Monday, November 11, 13
Scaling Pinterest
Evolution
Monday, November 11, 13
Scaling Pinterest
March 2010
Growth
Page views per day
Mar 2010
Jan 2011 Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
March 2010
Growth
Page views per day
Mar 2010
Jan 2011 Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
March 2010
Growth
·
RackSpace
·
1 small Web Engine
·
1 small MySQL DB
·
1 Engineer + 2 Founders
Page views per day
Mar 2010
Jan 2011 Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
March 2010
Growth
Monday, November 11, 13
Scaling Pinterest
March 2010
Growth
Monday, November 11, 13
Scaling Pinterest
January 2011
Growth
Mar 2010
Jan 2011
Jan 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
January 2011
Growth
Mar 2010
Jan 2011
Jan 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
January 2011
Growth
·
Amazon EC2 + S3 +
CloudFront
·
1 NGinX, 4 Web Engines
·
1 MySQL DB + 1 Read Slave
·
1 Task Queue + 2 Task
Processors
·
1 MongoDB
·
2 Engineers + 2 Founders
Mar 2010
Jan 2011
Jan 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
Monday, November 11, 13
Scaling Pinterest
September 2011
Growth
Mar 2010
Jan 2011
Jan 2012 May 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
September 2011
Growth
Mar 2010
Jan 2011
Jan 2012 May 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
September 2011
Growth
·
Amazon EC2 + S3 + CloudFront
·
2 NGinX, 16 Web Engines + 2 API
Engines
·
5 Functionally Sharded MySQL DB +
9 read slaves
·
4 Cassandra Nodes
·
15 Membase Nodes (3 separate
clusters)
·
8 Memcache Nodes
·
10 Redis Nodes
·
3 Task Routers + 4 Task Processors
·
4 Elastic Search Nodes
·
3 Mongo Clusters
·
3 Engineers (8 Total)
Mar 2010
Jan 2011
Jan 2012 May 2012
Page views per day
Monday, November 11, 13
Scaling Pinterest
It will fail. Keep it simple.
Monday, November 11, 13
Scaling Pinterest
If you’re the biggest user of a
technology, the challenges will
be greatly amplified
Monday, November 11, 13
Scaling Pinterest
January 2012
Growth
Monday, November 11, 13
Scaling Pinterest
April 2012
Growth
Mar 2010
Page views per day
Mar 2010
Jan 2011
Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
April 2012
Growth
Mar 2010
Page views per day
Mar 2010
Jan 2011
Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
April 2012
Growth
Mar 2010
·
Amazon EC2 + S3 + Edge Cast
·
135 Web Engines + 75 API Engines
·
10 Service Instances
·
80 MySQL DBs (m1.xlarge) + 1 slave
each
·
110 Redis Instances
·
60 Memcache Instances
·
2 Redis Task Manager + 60 Task
Processors
·
3rd party sharded Solr
Page views per day
Mar 2010
Jan 2011
Jan 2012 May 2012
Monday, November 11, 13
Scaling Pinterest
April 2012
Growth
Mar 2010
Page views per day
Mar 2010
Jan 2011
Jan 2012 May 2012
·
12 Engineers
·
1 Data Infrastructure
·
1 Ops
·
2 Mobile
·
8 Generalists
·
10 Non-Engineers
Monday, November 11, 13
Scaling Pinterest
Scaling Pinterest
Monday, November 11, 13
Scaling Pinterest
April 2013
Growth
Page views per day
April 2012
April 2013
Monday, November 11, 13
Scaling Pinterest
April 2013
Growth
Page views per day
April 2012
April 2013
Monday, November 11, 13
Scaling Pinterest
April 2013
Growth
·
Amazon EC2 + S3 + Edge Cast
·
400+ Web Engines + 400+ API
Engines
·
70+ MySQL DBs (hi.4xlarge on SSDs)
+ 1 slave each
·
100+ Redis Instances
·
230+ Memcache Instances
·
10 Redis Task Manager + 500 Task
Processors
·
65+ Engineers (130+ total)
Page views per day
April 2012
April 2013
Monday, November 11, 13
Scaling Pinterest
April 2013
Growth
·
Amazon EC2 + S3 + Edge Cast
·
400+ Web Engines + 400+ API
Engines
·
70+ MySQL DBs (hi.4xlarge on SSDs)
+ 1 slave each
·
100+ Redis Instances
·
230+ Memcache Instances
·
10 Redis Task Manager + 500 Task
Processors
·
65+ Engineers (130+ total)
Page views per day
April 2012
April 2013
·
8 services (80 instances)
·
Sharded Solr
·
20 HBase
·
12 Kafka + Azkabhan
·
8 Zookeeper Instances
·
12 Varnish
Monday, November 11, 13
Scaling Pinterest
April 2013
Growth
Page views per day
April 2012
April 2013
·
65+ Engineers
·
7 Data Infrastructure + Science
·
7 Search and Discovery
·
9 Business and Platform
·
6 Spam, Abuse, Security
·
9 Web
·
9 Mobile
·
2 growth
·
10 Infrastructure
·
6 Ops
·
65+ Non-Engineers
Monday, November 11, 13
Scaling Pinterest
Monday, November 11, 13
Scaling Pinterest
Technologies
Monday, November 11, 13
Scaling Pinterest
ELB
Routing & Filtering
(Varnish)
All connection pairings managed by ZooKeeper
Puppet
StatsD
Monit
Sensu
Web App
(Python)
API App
(Python / JS / HTML)
Task Processing
(Python/Pyres)
MySQL Service
(Java/Finagle)
Memcache Mux
(Nutcracker)
Follower Service
(Python/Thrift)
Feed Service
(Python/Thrift)
Images
(S3 + CDN)
Sharded
MySQL
Memcache
Redis
HBase
Task Queue
(Redis)
Search Service
(Python/Thrift)
Spam Service
(Python/Thrift)
Arch
Overview
Monday, November 11, 13
Scaling Pinterest
Data
Pipeline
Tripwire (Spam)
Qubole
S3
API App
(Python)
Task Processing
(Python/Pyres)
Kafka
S3 Copier
Pinball
Web App
(Python)
Redshift
Monday, November 11, 13
Scaling Pinterest
Web App
NGinX
Website Rendering (x8)
(Python / JS / HTML)
API
Monday, November 11, 13
Scaling Pinterest
Our MySQL Sharding?
http://www.infoq.com/presentations/Pinterest
Monday, November 11, 13
Scaling Pinterest
Choosing
Your
Tech
Monday, November 11, 13
Scaling Pinterest
Hosting
Why Amazon Web Services (AWS)?
Variety of servers running Linux
Very good peripherals: load balancing, DNS, map
reduce, basic security, and more
Good reliability
Very active dev community
Not cheap, but...
Monday, November 11, 13
Scaling Pinterest
Hosting
Why Amazon Web Services (AWS)?
Variety of servers running Linux
Very good peripherals: load balancing, DNS, map
reduce, basic security, and more
Good reliability
Very active dev community
Not cheap, but...
New instances ready in seconds
Monday, November 11, 13
Scaling Pinterest
Hosting
AWS Usage
Route 53 for DNS
ELB for 1st tier load balance
EC2 Ubuntu Linux
Varnish layer
All web, API, background appliances
All services
All databases and caches
S3 for images, logs
Monday, November 11, 13
Scaling Pinterest
Code
Why Python?
Extremely mature
Well known and well liked
Solid active community
Very good libraries specifically targeted to web
development
Effective rapid prototyping
Open Source
Monday, November 11, 13
Scaling Pinterest
Code
Why Python?
Extremely mature
Well known and well liked
Solid active community
Very good libraries specifically targeted to web
development
Effective rapid prototyping
Open Source
Some Java and Go...
Faster, lower variance response time
Monday, November 11, 13
Scaling Pinterest
Code
Python Usage
All web backend, API, and related business logic
Most services
Monday, November 11, 13
Scaling Pinterest
Code
Python Usage
All web backend, API, and related business logic
Most services
Java and Go Usage
Varnish plugins
Search indexers
High frequency services (e.g., MySQL service)
Monday, November 11, 13
Scaling Pinterest
Production
Data
Why MySQL and Memcache?
Extremely mature
Well known and well liked
(MySQL) Rarely catastrophic loss of data
Response time to request rate increases linearly
Very good software support: XtraBackup, Innotop, Maatkit
Solid active community
Open Source
Monday, November 11, 13
Scaling Pinterest
Production
Data
MySQL and Memcache Usage
Storage / Caching of core data
Users, boards, pins, comments, domains
Mappings (e.g., users to boards, user likes, repin info)
Legal compliance data
Monday, November 11, 13
Scaling Pinterest
Why Redis?
Well known and well liked
Active community
Consistently good performance
Variety of convenient and efficient data structures
3 Flavors of Persistence: Now, Snapshot, Never
Open Source
Production
Data
Monday, November 11, 13
Scaling Pinterest
Redis Usage
Follower data
Configurations
Public feed pin IDs
Caching of various core mappings (e.g., board to pins)
Production
Data
Monday, November 11, 13
Scaling Pinterest
Why HBase?
Small, but growing loyal community
Difficult to hire for, but...
Non-volatile, O(1), extremely fast and efficient storage
Strong Hadoop integration
Consistently good performance
Used by Facebook (bigger than us)
Seems to work well
Open Source
Production
Data
Monday, November 11, 13
Scaling Pinterest
HBase Usage
User feeds (pin IDs are pushed to feeds)
Rich pin details
Spam features
User relationships to pins
Production
Data
Monday, November 11, 13
Scaling Pinterest
What happened to Cassandra,
Mongo, ES, and Membase?
Production
Data
Does it meet your needs?
How mature is the product?
Is it commonly used? Can you hire people who have used it?
Is the community active? Can you get help?
How robust is it to failure?
How well does it scale? Will you be the biggest user?
Does it have a good debugging tools? Profiler? Backup
software?
Is the cost justified?
Monday, November 11, 13
Scaling Pinterest
A 2nd chance...
Monday, November 11, 13
Scaling Pinterest
Stu we could have done better
Logging on day 1 (StatsD, Kafka, Map Reduce)
Log every request, event, signup
Basic analytics
Recovery from data corruption or failure
Alerting on day 1
A 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Stu we could have done better
Shard our MySQL storage much earlier
Once you start relying on read slaves, start the
timebomb countdown
We also fell into the NoSQL trap (Membase,
Cassandra, Mongo, etc)
Pyres for background tasks day 1
Hire technical operations eng earlier
Chef / Puppet earlier
Unit testing earlier (Jenkins for builds)
A 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Stu we could have done better
A/B testing earlier
Decider on top of Zookeeper WATCH
Progressive roll out
Kill switches
A 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Looking Forward
Continually improve Pinner experience
Help Pinners discover more of the things they love
Better uptime and lower latency
Faster development times
Reduce spam and abuse
Continually improve collaboration and build bigger,
better, faster products
180 Pinployees and beyond
What’s
next?
Monday, November 11, 13
Scaling Pinterest
Have fun
Monday, November 11, 13
marty@pinterest.com
pinterest.com/martaaay
yashh@pinterest.com
pinterest.com/yashh
Monday, November 11, 13
marty@pinterest.com
pinterest.com/martaaay
yashh@pinterest.com
pinterest.com/yashh
Monday, November 11, 13
Scaling Pinterest
If I could do it all over again...
Stronger ACID transactional guarantees across multiple
systems
Currently have: sometimes A, best effort C, I, D,
no silent failure
Want: sometimes A, eventual C, I, D, no silent
completion
My 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Transactional tasks
All tasks become a dependency tree of repeatable
synchronous or asynchronous actions
All actions must be repeatable
Otherwise, must add repeatability
All tasks get a unique transaction number
Counters are tricky
My 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Transactional tasks
All tasks become a dependency tree of repeatable
synchronous or asynchronous actions
Sync actions are executed in order
Async actions are executed in any order
Repeat until successful or too many failures
Too many failures -> put in per task failure queue
Gives eventual C, I, D
No silent completion and A require extra effort
My 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Transactional tasks example
Pin create sync
Write empty pin object
Write pin ID to board, likes, user’s pins, clear caches
Write pin object
Pin not shown until pin object created -> Atomicity!
My 2nd
Chance
Monday, November 11, 13
Scaling Pinterest
Transactional tasks example
Pin create async
Write pin to required user feeds and public feeds
Feeds are sorted sets. Reinsertion is okay.
Send emails, Facebook Likes, Twitter Tweets
Before send, check / record in temporary storage
-> Gives (temporary) repeatability
My 2nd
Chance
Monday, November 11, 13